TELLING THE T&E STORY USING ANALYTICS-
BASED NARRATIVE VISUALIZATION
Authors Painter, Michael K.; Madanagopal, Karthic; Swaminathan,
Kannan; Jones, Charles H.
Publisher International Foundation for Telemetering
Journal International Telemetering Conference Proceedings
Rights Copyright © held by the author; distribution rights International
Foundation for Telemetering
Download date 06/12/2023 06:01:21
Link to Item http://hdl.handle.net/10150/631698
DISTRIBUTION STATEMENT A. Approved for public release; Distribution is unlimited 412TW-PA-18341
TELLING THE T&E STORY USING ANALYTICS-BASED
NARRATIVE VISUALIZATION
Michael K. Painter, P.E.
Karthic Madanagopal
Kannan Swaminathan
Knowledge Based Systems, Inc.
College Station, TX 77840
mpainter@kbsi.com
Charles H. Jones, PhD
C. H. Jones Consulting, LLC
Eugene, OR 97405
chjonesconsulting@gmail.com
ABSTRACT
There continues to be growing pressure to sell off spectrum currently allocated for defense
purposes in favor of private sector applications, prompting concerns that we will soon
reach a point where Department of Defense (DoD) needs can no longer be met. In
response, the Range Commanders Council (RCC) Frequency Management Group (FMG)
developed a baseline set of standard metrics to measure spectrum utilization, demand,
efficiency, and operational effectiveness. Using this standard (RCC 707-14) as a foundation,
a Spectrum Management Metrics Toolkit (SMMT) has been developed to calculate, plot, and
display these metrics. The challenge now is leveraging these metrics to inform and
construct the arguments needed to maintain access to needed spectrum.
The purpose of this paper is to describe progress toward the development of a
methodology and a set of analytics based on the RCC standard to build such a compelling
narrative. The methodology is based on a data analytics and communication concept,
called “Story Points,” which seeks to guide users in the discovery, composition, and delivery
of targeted narratives and supporting graphics derived through mining available data
sources.
INTRODUCTION
Today’s military operations increasingly rely on the ability to maintain full access and
reliable control of the radio frequency (RF) spectrum for communications, radar, electronic
warfare, remote fires, avionics, global positioning, logistics, medical support, test and
evaluation (T&E), training, and signals intelligence uses. Meanwhile, there is
correspondingly rapid growth in demand for commercial broadband wireless services and
other commercial uses requiring access to RF spectrum. Unquestionably, access to
spectrum is central to a wide range of business and consumer communication, research
and development (R&D) and information technology (IT) purposes, such as private and
public telecommunication operations (e.g., mobile phone networks, wireless Internet
communication, aviation, shipping, defense, public safety), broadcasting, radar, astronomy
DISTRIBUTION STATEMENT A. Approved for public release; Distribution is unlimited 412TW-PA-18341
and various other applications including countless short-range, low-power wireless
devices [1]. Contention for access to RF spectrum has led to an ongoing series of spectrum
auctions allowing commercial concerns to acquire licenses to spectrum that has historically
been allocated for federal use.
Despite concerted efforts to retain control of key bands of the spectrum, the prospect of
relieving strained federal budgets continues to drive further sell-offs. Even more
problematic has been the perception that the DoD is an inefficient user of the spectrum,
doesn’t need all it has, and is unjustifiably hoarding spectrum.
That narrative has gained traction, in part, by exploiting the tendency to confuse
“efficiency” with “utilization.” Utilization is the proportion of the available time that a
resource is being used or is in operation, usually expressed as a percentage.
1
The
argument essentially asserts that if music’s not playing (analogously speaking) at every
point on the dial 24 hours a day, the spectrum is not being used efficiently. In effect, higher
activity is equated with higher productivity. Given this logic, one could argue that leaving a
fire truck parked in the firehouse, as opposed to driving it continuously, is inefficient. An
equivalent T&E situation is reserving, but not having to issue a flight termination command
during on a missile test.
To be efficient means “productive of the desired effects [with minimal] waste.”
2
Efficiency,
therefore, should be judged in the context of the “desired effects.” For those seeking to
leverage spectrum for commercial broadband services, the desired effect is to maximize
profits for the company’s shareholders. In contrast, the desired effect of the T&E and
training communities is deterrence through developing and sustaining superior
warfighting capabilities. The two clearly can’t be easily compared.
The basic challenge is one of deciding how to best use a public resource weighing the
relative importance of each competing application. It is like deciding whether that prime
piece of real estate in a city’s main shopping district should be used for a firehouse or for
that office building proposed by a major investor, thus becoming a balancing act weighing
potential economic gains versus the risk of life and property loss. Or it’s like deciding
whether the best use of the Redwoods national forest is as a tree farm versus as a
monument to natural beauty and a protected habitat for rare species.
Ultimately, if the DoD hopes to maintain access to the spectrum it needs for T&E and
training purposes—whether for exclusive use or shared—it will have to change the
narrative that now frames the debate. Minimally, that narrative must establish the DoD’s
need for assured access to the spectrum, communicate the impacts of further restricting
1
For example, in the depot maintenance, repair, and overhaul (MRO) domain, technician efficiency is defined
as standard hours earned building to inventory divided by the total number of shift hours available. Pressure
to maintain high efficiency as defined here gives rise to unintended behaviors, including over-production
(resulting in excess inventory), cherry picking, and sandbagging.
2
https://www.merriam-webster.com/dictionary/efficient
DISTRIBUTION STATEMENT A. Approved for public release; Distribution is unlimited 412TW-PA-18341
access, and demonstrate responsible stewardship in the use of the spectrum. Much of the
difficulty in doing so in the past stems from the lack of well-defined metrics and tools to:
Accurately estimate current and future spectrum needs;
Account for actual versus scheduled utilization of the spectrum that is allocated, so as to
demonstrate responsible stewardship of the spectrum; and
Quantify the cost and schedule implications of the loss of needed spectrum.
The Spectrum Efficiency Through Metrics (SETM) effort directly supports these goals by
applying frequency metrics standards to monitor, assess, and improve the efficient use of
spectrum while simultaneously working to give leaders the tools needed to defend
continued access to T&E spectrum. The effort leverages many years’ worth of research
invested to define a baseline set of standard metrics for spectrum utilization, demand,
efficiency, and operational effectiveness. These metrics are formalized in the RCC FMG
Spectrum Management Metrics standard [2].
SPECTRUM MANAGEMENT METRICS TOOLKIT OVERVIEW
Perhaps the most visible product of the SETM effort will be a set of tools designed to
calculate, plot, and display frequency management metrics. Collectively, we refer to this set
of tools as the Spectrum Management Metrics Toolkit (SMMT).
Based on the metrics defined in RCC-707-14, the SMMT is designed to support calculation
and display of frequency management metrics in several categories, including: Spectrum
Occupancy, Utilization, Efficiency, Frequency Scheduling Operational Metrics, Predictive
and “What-if” Metrics.
Figure 1. Sampling of SMMT Metrics
For further information on this toolkit, see [3],
TELLING THE T&E STORY USING ANALYTICS
Metrics like those in the RCC standard can be powerful in and of themselves. They become
even more so when presented and embellished as part of a narrative that tells a compelling
story or argument. Part of the SETM research seeks to surface better ways to organize
DISTRIBUTION STATEMENT A. Approved for public release; Distribution is unlimited 412TW-PA-18341
metrics displays individually or in combination to support two goals: (i) highlight
important observations in the data to facilitate better management of spectrum, and (ii)
surface and better communicate spectrum needs and the impact of inaccessibility to that
spectrum.
With regard to spectrum management, the question being addressed is, “How can we apply
the metrics to surface opportunities to improve the effective use of spectrum?” For
example, users may wish to display metrics side-by-side to investigate possible causal
relationships leading to more efficient spectrum use. With regard promoting better access
to needed spectrum, the question is, “How can we organize the display of the metrics to
compose and communicate compelling narratives?” Using an example constructed from
teacher turnover statistics spanning multiple years (Figure 2), we identified several
constructs that would be useful in building such a narrative (Figure 3).
Figure 2. Narrative Visualization Example – Teacher Turnover
Figure 3. Narrative Visualization Composition Elements
Narrative elements may include (i) comparison to a benchmark value; (ii) identification of
a potential trend or cluster membership; (iii) a value judgment (e.g., observed trend is
positive); (iv) a definition of the scope or range of the sets involved; (v) the prognosis (i.e.,
likely future outcome or impacts); and (vi) indications of potential causal factors.
The resulting methodology is based on a data analytics and communication concept, called
“Narrative Visualization” or “Story Points,”
3
which seeks to guide users in the discovery,
composition, and delivery of targeted narratives and supporting graphics derived through
mining available data sources.
3
See https://www.tableau.com/about/blog/2014/5/82-preview-tell-story-your-data-story-points-30761
DISTRIBUTION STATEMENT A. Approved for public release; Distribution is unlimited 412TW-PA-18341
Based on sample data collected thus far, we were able to construct several examples of
possible narratives, as described below. One purpose behind this exercise was to identify
the SMMT extensions needed to surface, evolve, store, display, and share such narratives.
A familiar example of the Narrative Visualization concept in the T&E community is shown
in Figure 4, which illustrates how one might annotate a single chart to draw attention to
key observations. The narrative visualization elements used in the example are indicated
by darkened versus obscured text at the bottom of the figure.
Figure 4. Increasing Data Requirements
Figure 5. UAVs Account for Significant Portion of Overall Spectrum Use
In Figure 5, one can see that MHz-hours (MH) scheduled by the various Combined Test
Force (CTF) users has been on a steady increase between 2001 and 2012. Taking a closer
DISTRIBUTION STATEMENT A. Approved for public release; Distribution is unlimited 412TW-PA-18341
look, we find that overall growth in spectrum demand was driven largely by demands
associated with unmanned aerial vehicle (UAV) testing (Figure 5).
Figure 6. F-35 Top Dog at EAFB
More recently, another major test program came on line—the F-35. The priority given to
the F-35 test program versus others is evident in the relative number of flights scheduled
and the number of frequency assignments made for the F-35. In 2017, for example, the
461
st
Flight Test Squadron (FLTS), which runs the F-35 test program, was by far the biggest
user (Figure 6). This figure might prompt the question, “Are the other CTFs asking for and
getting the frequency they need?” and “If not, how much will test program completion be
delayed?”
Figure 7. F-22 Completes Developmental Testing
DISTRIBUTION STATEMENT A. Approved for public release; Distribution is unlimited 412TW-PA-18341
In the example shown above, the F-35 program was involved in the developmental test
phase whereas most of the other test programs had transitioned into their follow-on
testing phase. When a program shifts from developmental testing to its follow-on testing
phase, the number of test flights and the amount of telemetry frequency required can drop
significantly. This pattern is evident in the example shown in Figure 7 for the F-22.
Given that there are multiple test programs underway at any given time, what can be
expected in the way of their collective telemetry frequency needs? Figure 8 indicates that
peak demand for spectrum increased year over year, and sometimes at a dramatic pace. In
2007, for example, peak demand at Edwards Air Force base (EAFB) appeared to be on track
to double every nine years. The Navy’s Echo range, on the other hand, looked like it might
easily double every three years.
Figure 8. Spectrum Demand Rising for Follow-on Testing
Figure 9. Spectrum Assignments Double While MHs per Frequency Assignment Are Cut by a Third
DISTRIBUTION STATEMENT A. Approved for public release; Distribution is unlimited 412TW-PA-18341
Earlier, we saw that between the years of 2001 through 2012, the number of frequency
requests and the demand for telemetry frequency was on a steady, albeit somewhat flat
climb. So, what happened during the intervening years? Based on the data shown in
Figure 9, looking only at the five-year period between 2012 and 2017, the total number of
spectrum assignments doubled while the total MHs increased by 25%. Meanwhile, the MHs
per frequency assignment was reduced by a third. This was caused by two factors. First,
the number of frequency requests per operation doubled. This may have been caused by
outfitting the test articles with more radios to send additional telemetry data. Second,
there was a drop in the number of hours flown per test mission, on average. F-35 missions,
for example, dropped their normal duration by half.
At some point, a steady increase in frequency needs will ultimately approach a saturation
point. It would appear that EAFB is nearing or has already reached that point (Figure 10).
In fiscal year 2017, the average MHz assigned per frequency request was 9.5 MHz. Given
that the total MHz available in the L- and S- bands was 285 MHz, the maximum number of
concurrent assignments at any one time was thirty. Based on a box plot showing the
number of concurrent assignments by time of day (from 7AM to 7PM), the actual number of
assignments from 10AM to noon exceeded this threshold more than a third of the time. In
terms of frequency, the total MHz scheduled exceeded what was available roughly 25% of
weekdays between 10AM and noon. The shortfall in spectrum appears to have been
accommodated for by flying on weekends and shifting to the C-band.
Figure 10. EAFB Approaching Saturation Point
One consequence of a spectrum loss and further crowding is a reduction in the number of
test operations that can be accommodated each day coupled with a lengthening of test
program schedules. This situation is illustrated by the example shown in Figure 11. In this
case, testing operations conducted between 2010 and 2012 at one of the DoD’s test ranges
were accomplished using much of the allotted spectrum in the L- and S-bands. By 2016, an
increase in bit rate requirements for these test programs—necessitated largely by the need
for high definition video—effectively cut in half the number of simultaneous test
DISTRIBUTION STATEMENT A. Approved for public release; Distribution is unlimited 412TW-PA-18341
operations that could be supported. Adding shifts to cover more than the typical 0600-
1800 day might be a partial solution, but is constrained by the need for good lighting for
the cameras. Even a change in modulation methods from analog FM to SOPQSK would have
little impact in this case. Without other solutions, the calendar time it takes to complete
test programs of this type could easily double, thereby undercutting our ability to stay
ahead of our adversaries.
Figure 11. Continued Increases in Bit Rate Requirements May Lengthen Test Schedules
Having studied how to compose these kinds of narratives, the task now involves defining
and building extensions to the SMMT to support their construction and maintenance.
CONCLUSIONS
Assured access to electronic spectrum is essential to the success of U.S. military operations
both now and in the future. T&E spectrum needs are a vital component of that
requirement. It is in that domain of spectrum application that we develop and maintain a
decisive technological advantage. The two are inextricably linked. As more advanced
technologies emerge, T&E spectrum needs will continue to grow. Many project that growth
to be exponential. Meanwhile, the more crowded the spectrum becomes, the more it will
become necessary to spread testing missions in time. Of course, doing so lengthens the
development time between new generations of fielded weapons technology. Longer lead
times, in turn, make it easier for enemy concerns to close the gap and achieve technological
parity, or worse.
As evidenced by the past, economic pressures to increase the scope of commercial
spectrum use will continue to drive further reallocations of DoD spectrum. If the DoD
hopes to maintain the spectrum it needs for T&E purposes, it will have to clearly
DISTRIBUTION STATEMENT A. Approved for public release; Distribution is unlimited 412TW-PA-18341
demonstrate both the need for that spectrum and the responsible, efficient use of the
spectrum. It must also dramatically improve its ability to quantitatively establish the
technical, cost, schedule, and safety implications of reduced access and control.
The emergence of the RCC’s frequency management metrics standard coupled with tools
like the one presented in this paper seek to provide the means to better manage and defend
needed T&E spectrum.
BIBLIOGRAPHY
[1] Ben Freyens, “The Economics of Spectrum Management: A Review”, School of
Economics, Australian National University, 2007.
[2] RCC 707-14 Spectrum Management Metrics Standards, Secretariat, Range Commanders
Council, White Sands Missile Range, Mew Mexico.
[3] Painter, M., Fernandes, R., Jones, C., and Madanagopal, K., “Defending T&E Spectrum
Through Automated Frequency Management Metrics Calculation,” Proceedings of the
International Telemetering Conference, 2017.
[4] Coping with Change: Managing RF Spectrum to Meet DoD Needs, Report of the Defense
Science Board Task Force on DoD Frequency Spectrum Issues, Office of the Under
Secretary of Defense for Acquisition and Technology, Washington, D.C., November 2000.
[5] Ernst, Darrell E., Carolyn A. Kahn, and David L. Portigal, “The Economic Importance of
Adequate Aeronautical Telemetry Spectrum,” The MITRE Corporation, February 2007.
[6] Report of the Defense Science Board 1996 Summer Study Task Force on Tactics and
Technology for 21st Century Military Superiority, Office of the Secretary of Defense,
October 1996.
[7] Chalfant, Timothy A. Telemetry Spectrum Encroachment: Taking Steps to Ensure the
Future, in Proceedings of the Systems Engineering, Test & Evaluation Conference, October
2002.
[8] Ernst, D., Hoh, Y., Portigal, D., “The Growth of Data Rates for Aeronautical Telemetering
and the Implications for the Radio Spectrum,” MITRE Technical Report MTR
04W0000032, May 2004.
Acknowledgment of Support: This project is funded by the Test Resource Management Center
(TRMC) Test and Evaluation/Science & Technology (T&E/S&T) Program through the U.S.
Army Program Executive Office for Simulation, Training, and Instrumentation (PEO STRI)
under Contract No. W900KK-16-C-0018. The Executing Agent and Program Manager work out
of the AFTC.
Disclaimer: Any opinions, findings and conclusions or recommendations expressed in this
material are those of the author(s) and do not necessarily reflect the views of the Test Resource
Management Center (TRMC) and Evaluation/Science & Technology (T&E/S&T) Program
and/or the U.S. Army Program Executive Office for Simulation, Training, & Instrumentation
(PEO STRI).
Public Affairs Clearance: 412TW-PA-18341: Telling the T&E Story Using Analytics-based
Narrative Visualization
